A Framework for Measuring Differences in Data Characteristics

A data mining algorithm builds a model that captures interesting aspects of the underlying data. We develop a framework for quantifying the difference, called the deviation, between two datasets in terms of the models they induce. In addition to being a quantitative, intuitively interpretable measure of difference, the deviation between two datasets can also be computed very fast. Our framework covers a wide variety of models including frequent itemsets, decision tree classifiers, and clusters, and captures standard measures of deviation such as the misclassification rate and the chi-squared metric as special cases. We also show how statistical techniques can be applied to the deviation measure to assess whether the difference between two models Contact author. Supported by a Microsoft Graduate Fellowship. y This research was supported by Grant 2053 from the IBM corporation. z Supported by Army Research Office grant DAAG55-98-1-0333.